[History server] Update history server examples and defaults by chiayi · Pull Request #5080 · ray-project/kuberay

chiayi · 2026-08-05T00:23:04Z

Why are these changes needed?

This PR updates the event port default to 8084 from 8080 which will clash with dashboard port. Along with changing the default, this PR also update examples to use the env var instead of the flags. And since the entrypoint for the collector and historyserver is not part of the image, we no longer need the command.

It was also found that RAY_ROOT_DIR was not added to historyserver, this PR adds it.

Edit: Update scope

Related issue number

Part of #4187

Labels

If this PR has user-facing changes that require documentation updates at release time, I have added the doc-updates-required label.
If this PR contains breaking changes, I have added the breaking-change label.

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

Manual test instructions

Ran the yaml against a live kubernetes cluster.

chiayi · 2026-08-05T00:43:32Z

The one line change in the ray controller unit test was to fix:

andrewsykim · 2026-08-05T00:44:54Z

          - name: RAY_DASHBOARD_AGGREGATOR_AGENT_EVENTS_EXPORT_ADDR
            value: "http://localhost:8084/v1/events"
-          - name: RAY_DASHBOARD_AGGREGATOR_AGENT_EXPOSABLE_EVENT_TYPES
+          - name: RAY_DASHBOARD_AGGREGATOR_AGENT_PUBLISHER_HTTP_ENDPOINT_EXPOSABLE_EVENT_TYPES


why this change?

IIRC this is just a variable update starting from 2.55+ to tell which event to export.

I think the change was reverted? ray-project/ray#59965
We should keep RAY_DASHBOARD_AGGREGATOR_AGENT_EXPOSABLE_EVENT_TYPES here

andrewsykim · 2026-08-05T00:45:13Z

                    TASK_PROFILE_EVENT,DRIVER_JOB_DEFINITION_EVENT,DRIVER_JOB_LIFECYCLE_EVENT,
                    ACTOR_DEFINITION_EVENT,ACTOR_LIFECYCLE_EVENT,NODE_DEFINITION_EVENT,NODE_LIFECYCLE_EVENT"
-          image: rayproject/ray:2.52.0
+          image: rayproject/ray:2.55.0


Use 2.56 while we're at it?

win5923 · 2026-08-05T05:59:37Z

Just wondering should we still keep this sample yaml or just change to use spec.historyServerOptions.collectorOptions instead, so the user don't need to manually set the envs and args.

This requires RayClusterHistoryServer feature gate. I think we could have a separate sample for this?

kuberay/ray-operator/controllers/ray/common/pod.go

Line 304 in 3dd6ce6

if features.Enabled(features.RayClusterHistoryServer) && instance.Spec.HistoryServerOptions != nil && instance.Spec.HistoryServerOptions.CollectorOptions != nil {

Got it, we already have one in the ray-operator samples.

https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.historyserver.yaml

machichima · 2026-08-05T08:00:03Z

        - name: collector
-          image: collector:v0.1.0
+          image: quay.io/kuberay/collector:nightly
          imagePullPolicy: IfNotPresent


If we use nightly, I think we need to set imagePullPolicy: Always, otherwise it will never pull the newest image. Also for other files.
Actually I think we can come back and update this after release to pin the specific version

machichima · 2026-08-05T08:03:30Z

Could we also remove command here?

Oh you know what, good point. So the env variables have not been converted on the history server side. History server will need to check for the RAY_ROOT_DIR env var. And I think it's small enough to add the change here (this PR) as well.

FYI: @andrewsykim

machichima · 2026-08-05T08:05:56Z

We'll need to update the ray docs after this is merge. Can do it together with #5085

machichima · 2026-08-05T08:10:17Z

CI failed with unable to start container process: error during container init: exec: \"--ray-cluster-namespace=test-ns-c5rxh\": executable file not found in $PATH"}. I think it's related to removing command from the YAML.

Future-Outlier

Hi, @chiayi
plz merge this PR #5082 before you get this PR merged, tks

Future-Outlier

I am thining that we should combine historyserver/config/raycluster.yaml and historyserver/config/rayjob.yaml to RayJob example, WDYT?

and also these 2 files might be better if we turn then to rayjob example too.
historyserver/config/raycluster-azureblob.yaml
historyserver/config/raycluster-gcs.yaml

andrewsykim · 2026-08-05T15:02:55Z

+1 to use ephemeral RayJob with spec.shutdownAfterJobFinishes or deletion policy API where it makes sense

chiayi · 2026-08-05T17:35:20Z

Hi, @chiayi plz merge this PR #5082 before you get this PR merged, tks

I think we can maybe combine that PR with this one? I can change it to all here and this PR already have the version update.

chiayi · 2026-08-05T17:52:06Z

and also these 2 files might be better if we turn then to rayjob example too.
historyserver/config/raycluster-azureblob.yaml
historyserver/config/raycluster-gcs.yaml

Will convert these to rayjobs

chiayi · 2026-08-05T18:14:52Z

So historyserver/config/raycluster-azureblob.yaml is hard coded into the e2e tests and I think it will require more additional code changes. I think I will leave it as is as I am not familiar with azure configurations,

chiayi · 2026-08-05T18:46:54Z

Some of these change leading to more changes and bloating this PR. I will not be changing the image and the azureblob yaml to avoid adding to this PR.

FYI: @machichima @andrewsykim

andrewsykim · 2026-08-05T19:07:23Z

+          containers:
+          - env:
+            - name: RAY_TMP_ROOT
+              value: &rayTmpRoot /tmp/ray


we should add a comment at the top of this file that it must be run with envsubst with an example

andrewsykim · 2026-08-05T19:08:07Z

+            - name: STORAGE_BACKEND
+              value: "gcs"
+            - name: RAY_ROOT_DIR
+              value: "log"


This one seems a bit odd, why do we use "log" for RAY_ROOT_DIR

Honestly do not remember the reason for setting root dir to logs. But we can probably just change it to root

I don't think root is a valid value for RAY_ROOT_DIR either, what's the default value if we don't set it? If that works just don't set it here?

andrewsykim · 2026-08-06T19:12:21Z

+          - name: STORAGE_BACKEND
+            value: "gcs"
+          - name: RAY_ROOT_DIR
+            value: "root"


I don't think "root" will work either? Is RAY_ROOT_DIR referring to Ray's root directory or the root directory used in storage? If it's storage, this variable name is confusing.

It's referring to the root directory in storage. So maybe STORAGE_ROOT_DIR instead?

yeah I would rename to STORAGE_ROOT_DIR. Also when it's empty it should just use the specified bucket's root dir

@chiayi please fix this in a fast follow-up

andrewsykim · 2026-08-06T19:17:05Z

+                  fieldPath: status.podIP
+            - name: FQ_RAY_IP
+              value: $(RAY_CLUSTER_NAME)-head-svc.$(RAY_CLUSTER_NAMESPACE).svc.cluster.local
+            - name: RAY_TMP_ROOT


Do we really need to set this? I think this will almost always be /tmp/ray

I think we have it as an option because it's also an option for rayclusters. In case users decide to change the raycluster temp-dir.

andrewsykim · 2026-08-06T19:18:42Z

+            - name: code-sample
+              mountPath: /home/ray/samples
+          - name: collector
+            image: collector:v0.1.0


use the official image?

This is so that the ci will use the PR's version of history server to run the tests. As @machichima mentioned here: #5080 (comment) we should separate it out so that there is a specific dir for test suite, but I think that should be a separate PR.

Ooops. correction, this is rayjob and not raycluster. And raycluster-gcs isn't currently used in e2e either.

machichima · 2026-08-07T03:57:47Z

            mountPath: *rayTmpRoot
        - name: collector
-          image: ${COLLECTOR_IMAGE}
+          image: collector:v0.1.0


We would need to unify here. Now we have two versions, one setting collector:v0.1.0 and the other set the nightly image (in historyserver/config/rayjob-gcs.yaml).
Are we going to:

keep collector:v0.1.0 here and update after release for the release version

use nightly version with image pull policy always

Have an env var for user to inject the image

What do you think about leaving it as is so ${COLLECTOR_IMAGE} and update once we release? so a combination of 3 and 1. So we don't change anything for the image here until release.

Sure, let's do this

Great! will do this!

machichima · 2026-08-07T05:32:27Z

+          imagePullSecrets:
+          affinity:


Why do we leave empty fields here?

Good question, so the raycluster part of the spec is actually copied over from raycluster.yaml. I guess they are currently just empty placeholders.

I see, should we just remove them?

machichima · 2026-08-07T05:34:30Z

+          - name: STORAGE_BACKEND
+            value: "gcs"
+          - name: RAY_ROOT_DIR
+            value: "storage-root"


I think there's mismatch in ray root dir. Here we use storage-root but in other places (e.g. historyserver/config/raycluster-azureblob.yaml) we use log

The log is actually hard coded in the e2e tests for azure and s3 and did not want to add to this PR even more. And I updated the raycluster-gcs because it is used in the ray doc/guide and did not want to cause confusion when users view them.

What we could do is when we open a RAY_ROOT_DIR fast followup change to STORAGE_ROOT_DIR, we can also make the changes to the e2e test as well? So no change here for the time being to avoid confusion.

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit aa941ee. Configure here.}

cursor · 2026-08-07T07:35:53Z

-                    TASK_PROFILE_EVENT,DRIVER_JOB_DEFINITION_EVENT,DRIVER_JOB_LIFECYCLE_EVENT,
-                    ACTOR_DEFINITION_EVENT,ACTOR_LIFECYCLE_EVENT,NODE_DEFINITION_EVENT,NODE_LIFECYCLE_EVENT"
-          image: rayproject/ray:2.52.0
+            value: "ALL"


Incomplete Ray 2.56 event export config

Medium Severity

After upgrading these samples to Ray 2.56.0 and switching exposable event types to ALL, only RAY_DASHBOARD_AGGREGATOR_AGENT_EXPOSABLE_EVENT_TYPES is set. The GCS samples updated in the same change also set RAY_DASHBOARD_AGGREGATOR_AGENT_PUBLISHER_HTTP_ENDPOINT_EXPOSABLE_EVENT_TYPES, and the removed comments previously called out that newer Ray needs that publisher variable for HTTP export to the collector.

Additional Locations (1)

historyserver/config/raycluster-azureblob.yaml#L30-L32

^{Reviewed by Cursor Bugbot for commit aa941ee. Configure here.}

Yes it should be RAY_DASHBOARD_AGGREGATOR_AGENT_EXPOSABLE_EVENT_TYPES instead of RAY_DASHBOARD_AGGREGATOR_AGENT_PUBLISHER_HTTP_ENDPOINT_EXPOSABLE_EVENT_TYPES @chiayi

machichima · 2026-08-07T08:58:05Z

+            - name: historyserver
+              mountPath: *rayTmpRoot
+          - name: collector
+            image: quay.io/kuberay/collector:nightly


Related to #5080 (comment)
I think we also need to update here?

machichima · 2026-08-07T09:50:33Z

+            - name: code-sample
+              mountPath: /home/ray/samples
+          - name: collector
+            image: quay.io/kuberay/collector:nightly


andrewsykim · 2026-08-07T13:10:55Z

+            imagePullPolicy: IfNotPresent
+            securityContext:
+              allowPrivilegeEscalation: true
+              privileged: true


why do we need to use priviledged containers? I really hope this is no longer required for Beta

andrewsykim · 2026-08-07T13:12:27Z

+            - name: historyserver
+              mountPath: *rayTmpRoot
+          tolerations:
+          - key: ray


this is a weird toleration to have here, do we need it?

chiayi · 2026-08-07T18:39:31Z

Followups:

Fast followup to update the RAY_ROOT_DIR to something like STORAGE_ROOT_DIR to avoid configuration confusion. Also set the value to storage-root from logs. This will require changing the hardcode in some of the tests.
Separate the samples directory to into samples/ and testdata/.
- So we no longer hardcode the images
- Sample no longer need to be v2.55
- Overall so that updating the samples won't break CI.

andrewsykim · 2026-08-07T22:10:13Z

+          - name: STORAGE_BACKEND
+            value: "gcs"
+          - name: RAY_ROOT_DIR
+            value: "storage-root"


For now can we just not set RAY_ROOT_DIR? In this case the default behavior should be using the root of the provided bucket. Or whichever subdirectory the user specifies in GCS_BUCKET gs://my-bucket/historyserver/

chiayi force-pushed the history-server branch from ad5ed54 to 2ba97ad Compare August 5, 2026 00:28

chiayi requested review from andrewsykim and machichima August 5, 2026 00:34

chiayi marked this pull request as ready for review August 5, 2026 00:34

chiayi force-pushed the history-server branch from 2ba97ad to ed22286 Compare August 5, 2026 00:40

chiayi requested review from MortalHappiness, kevin85421 and rueian as code owners August 5, 2026 00:40

andrewsykim reviewed Aug 5, 2026

View reviewed changes

machichima mentioned this pull request Aug 5, 2026

[History Server] Bump ray to 2.54.0 and use "ALL" for exposable event types #5082

Closed

machichima reviewed Aug 5, 2026

View reviewed changes

Comment thread historyserver/config/raycluster-azureblob.yaml Outdated

win5923 reviewed Aug 5, 2026

View reviewed changes

machichima reviewed Aug 5, 2026

View reviewed changes

Future-Outlier reviewed Aug 5, 2026

View reviewed changes

cursor Bot reviewed Aug 5, 2026

View reviewed changes

Comment thread historyserver/config/rayjob-gcs.yaml

chiayi force-pushed the history-server branch from 183fb20 to acca346 Compare August 5, 2026 18:24

cursor Bot reviewed Aug 5, 2026

View reviewed changes

Comment thread historyserver/config/raycluster.yaml

chiayi force-pushed the history-server branch from acca346 to 0dc3d50 Compare August 5, 2026 18:32

cursor Bot reviewed Aug 5, 2026

View reviewed changes

Comment thread historyserver/config/raycluster.yaml Outdated

chiayi force-pushed the history-server branch from 0dc3d50 to 58ff4c4 Compare August 5, 2026 18:43

andrewsykim reviewed Aug 5, 2026

View reviewed changes

chiayi added 6 commits August 6, 2026 16:59

Update history server examples

7895b02

update raycluster ray version for history server samples

fe35cd5

fix ci

65999ff

update ray version

4ee49e6

Update historyserver and convery raycluster-gcs to rayjob-gcs

e1501f1

Added 2.56 updated expected grafana health response

6e5efb1

chiayi force-pushed the history-server branch from 7044c38 to 9ec0fb0 Compare August 6, 2026 17:24

chiayi mentioned this pull request Aug 6, 2026

[History Server] Replace "runtime class name" flag and env var with "storage backend" #5085

Open

6 tasks

cursor Bot reviewed Aug 6, 2026

View reviewed changes

Comment thread historyserver/config/raycluster.yaml Outdated

rebase and update

b308655

chiayi force-pushed the history-server branch from 9ec0fb0 to b308655 Compare August 6, 2026 18:02

cursor Bot reviewed Aug 6, 2026

View reviewed changes

Comment thread historyserver/config/rayjob-gcs.yaml Outdated

reorder env var

bb1e2a7

andrewsykim reviewed Aug 6, 2026

View reviewed changes

add back raycluster-gcs example to not break doc

c9f2fd5

chiayi force-pushed the history-server branch from 13e3185 to c9f2fd5 Compare August 7, 2026 00:15

Update storage root env var value to make it clear what it is for

2c63b00

machichima reviewed Aug 7, 2026

View reviewed changes

chiayi force-pushed the history-server branch from ec33464 to 4486391 Compare August 7, 2026 07:21

Update gcs yaml

aa941ee

chiayi force-pushed the history-server branch from 4486391 to aa941ee Compare August 7, 2026 07:27

cursor Bot reviewed Aug 7, 2026

View reviewed changes

machichima reviewed Aug 7, 2026

View reviewed changes

andrewsykim reviewed Aug 7, 2026

View reviewed changes

chiayi added 2 commits August 7, 2026 17:28

remove unused configuration

f983057

Updated env var for event exporter

bda7aeb

chiayi force-pushed the history-server branch from 3a87d8c to bda7aeb Compare August 7, 2026 18:28

andrewsykim reviewed Aug 7, 2026

View reviewed changes

Uh oh!

Conversation

chiayi commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Labels

Checks

Manual test instructions

Uh oh!

chiayi commented Aug 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

machichima Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

win5923 Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chiayi Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

machichima commented Aug 5, 2026

Uh oh!

machichima commented Aug 5, 2026

Uh oh!

Future-Outlier left a comment

Choose a reason for hiding this comment

Uh oh!

Future-Outlier left a comment

Choose a reason for hiding this comment

Uh oh!

andrewsykim commented Aug 5, 2026

Uh oh!

chiayi commented Aug 5, 2026

Uh oh!

chiayi commented Aug 5, 2026

Uh oh!

chiayi commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chiayi commented Aug 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chiayi Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewsykim Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chiayi commented Aug 5, 2026 •

edited

Loading

machichima Aug 5, 2026 •

edited

Loading

win5923 Aug 5, 2026 •

edited

Loading

chiayi Aug 5, 2026 •

edited

Loading

chiayi commented Aug 5, 2026 •

edited

Loading

chiayi Aug 5, 2026 •

edited

Loading

andrewsykim Aug 6, 2026 •

edited

Loading